Skip to content

Add initial single-sandbox-executing-at-a-time support for Hypervisor.framework - #1674

Open
syntactically wants to merge 17 commits into
mainfrom
lm/hvf
Open

Add initial single-sandbox-executing-at-a-time support for Hypervisor.framework#1674
syntactically wants to merge 17 commits into
mainfrom
lm/hvf

Conversation

@syntactically

@syntactically syntactically commented Jul 21, 2026

Copy link
Copy Markdown
Member

No description provided.

Copilot AI review requested due to automatic review settings July 21, 2026 15:27
@syntactically syntactically added kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. ready-for-review PR is ready for (re-)review labels Jul 21, 2026
Copilot AI reviewed Jul 21, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds initial Hypervisor.framework (HVF) support to hyperlight-host on macOS, along with a snapshot ABI/version bump and memory-layout/page-size changes needed for platforms with 16K host pages.

Changes:

  • Introduce an HVF VirtualMachine backend and macOS-specific build/runtime glue (bindgen + codesigning runner + interrupt handling).
  • Bump snapshot ABI/goldens versions and update snapshot file metadata/config handling for the new hypervisor tag and ABI.
  • Adjust host/guest memory layout assumptions (page size usage, base addresses, alignment) and tune tests to avoid unsupported parallelism on HVF.

Reviewed changes

Copilot reviewed 39 out of 41 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
src/hyperlight_host/tests/snapshot_goldens/platform.rs Add HVF hypervisor tag handling in snapshot golden platform selection.
src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs Bump goldens version and set arch-specific compatibility list.
src/hyperlight_host/tests/sandbox_host_tests.rs Serialize/scale down thread-heavy tests under HVF; adjust scratch sizing.
src/hyperlight_host/tests/integration_test.rs Reduce stress-test thread counts under HVF.
src/hyperlight_host/src/sandbox/snapshot/tripwires.rs Bump expected snapshot ABI and update page-size/base-address invariants.
src/hyperlight_host/src/sandbox/snapshot/mod.rs Align snapshot memory to host page size; update tests to use runtime page size.
src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs Bump SNAPSHOT_ABI_VERSION for snapshot memory blob.
src/hyperlight_host/src/sandbox/snapshot/file/config.rs Add HVF hypervisor tag and macOS/aarch64 CPU vendor behavior.
src/hyperlight_host/src/sandbox/snapshot/file_tests.rs Accept hvf annotation/tag in snapshot file tests.
src/hyperlight_host/src/sandbox/initialized_multi_use.rs Use runtime page size in test helpers and overlap calculations.
src/hyperlight_host/src/sandbox/file_mapping.rs Broaden mmap-backed file mapping resource handling to unix.
src/hyperlight_host/src/sandbox/config.rs Enable interrupt retry delay config for HVF as well as Linux hypervisors.
src/hyperlight_host/src/mem/shared_mem.rs Generalize mmap code to unix and switch guard sizing to runtime page size.
src/hyperlight_host/src/mem/shared_mem_tests.rs Update tests to use runtime page size.
src/hyperlight_host/src/mem/mgr.rs Account for host page alignment when locating snapshot PT tail.
src/hyperlight_host/src/mem/memory_region.rs Use vmem page size constants and runtime page size where needed.
src/hyperlight_host/src/mem/layout.rs Move base address to 0x4000; align sizes for mixed guest/host page sizes.
src/hyperlight_host/src/hypervisor/virtual_machine/whp.rs Make register setters require &mut self to match new VM trait.
src/hyperlight_host/src/hypervisor/virtual_machine/mshv/x86_64.rs Make register setters require &mut self to match new VM trait.
src/hyperlight_host/src/hypervisor/virtual_machine/mod.rs Add HVF module/type and selection; extend error enums for HVF sync/errors.
src/hyperlight_host/src/hypervisor/virtual_machine/kvm/x86_64.rs Make register setters require &mut self to match new VM trait.
src/hyperlight_host/src/hypervisor/virtual_machine/kvm/aarch64.rs Make register setters require &mut self to match new VM trait.
src/hyperlight_host/src/hypervisor/virtual_machine/hvf/mod.rs New HVF backend implementing VM lifecycle, memory mapping, and state sync.
src/hyperlight_host/src/hypervisor/virtual_machine/hvf/fp_abi.c C ABI shims for SIMD FP register access not supported in stable Rust FFI.
src/hyperlight_host/src/hypervisor/virtual_machine/hvf/bindings.h Header for bindgen + shim declarations, includes Hypervisor.framework.
src/hyperlight_host/src/hypervisor/surrogate_process.rs Use runtime page size for guard pages in Windows surrogate mapping.
src/hyperlight_host/src/hypervisor/mod.rs Refactor interrupt handle state tracking; add HVF cancel mechanism support.
src/hyperlight_host/src/hypervisor/hyperlight_vm/x86_64.rs Switch to new interrupt handle constructors; adjust VM mutability.
src/hyperlight_host/src/hypervisor/hyperlight_vm/mod.rs Route interrupt state through the new state machine in the VM run loop.
src/hyperlight_host/src/hypervisor/hyperlight_vm/aarch64.rs Add HVF VM construction path and HVF interrupt handle wiring.
src/hyperlight_host/Cargo.toml Add hvf feature, bindgen/cc build deps, parking_lot, and default-enable HVF.
src/hyperlight_host/build.rs Generate HVF bindings and compile the C shim; add HVF cfg alias.
src/hyperlight_common/src/mem.rs Remove fixed page-size constants from mem (shifted to vmem).
src/hyperlight_common/src/arch/aarch64/layout.rs Adjust fixed AArch64 addresses to accommodate 16K pages.
flake.nix Add aarch64-darwin target support and adjust tooling availability per platform.
flake.lock Update pinned nixpkgs revision/hash.
dev/macos-sign-and-run.sh Add macOS runner that codesigns binaries with hypervisor entitlement before exec.
dev/macos-entitlements.plist Add com.apple.security.hypervisor entitlement for signed binaries.
CHANGELOG.md Document the snapshot incompatibility caused by the address/layout change.
Cargo.lock Record new dependencies (parking_lot, bindgen, cc) in the lockfile.
.cargo/config.toml Configure macOS Cargo runner to use the codesign wrapper script.

Comment thread src/hyperlight_host/src/hypervisor/mod.rs
Comment thread src/hyperlight_host/build.rs
Comment thread src/hyperlight_host/src/hypervisor/virtual_machine/mod.rs
Comment thread src/hyperlight_host/src/hypervisor/virtual_machine/mod.rs
Comment thread src/hyperlight_host/src/hypervisor/virtual_machine/mod.rs
Comment thread src/hyperlight_host/src/hypervisor/hyperlight_vm/mod.rs
@syntactically syntactically added the regen-goldens Regenerate snapshot golden fixtures label Jul 21, 2026
@siennathesane

siennathesane commented Jul 22, 2026 •
edited
Loading

Copy link
Copy Markdown

I submitted #1681, which has full surrogate support for parallel VMs. Would you be able to take a look at that before merging this?

(Also, your copyright is out of date in several files)

ludfjig
ludfjig reviewed Jul 23, 2026
View reviewed changes

@ludfjig ludfjig left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks really good. I haven't reviewed the changes to InterruptHandle yet. Btw do you know if just snapshot-goldens-pull will work on mac once golden snapshto images areadded? I haven't tested it but I think there mgiht some hardcoded stuff there

Comment thread src/hyperlight_host/src/hypervisor/hyperlight_vm/x86_64.rs
Comment thread src/hyperlight_host/src/mem/layout.rs
Comment thread src/hyperlight_host/src/mem/shared_mem.rs
Comment thread src/hyperlight_host/build.rs
Comment thread src/hyperlight_host/src/sandbox/snapshot/mod.rs
Comment thread src/hyperlight_host/src/sandbox/snapshot/mod.rs
@syntactically syntactically mentioned this pull request Jul 23, 2026
Open
jsturtevant
jsturtevant reviewed Jul 31, 2026
View reviewed changes

@jsturtevant jsturtevant left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing work! We should be able to run this on CI now?

Comment thread src/hyperlight_host/src/hypervisor/virtual_machine/hvf/mod.rs
Comment thread src/hyperlight_host/src/hypervisor/virtual_machine/hvf/mod.rs Outdated
jsturtevant
jsturtevant reviewed Jul 31, 2026
View reviewed changes
Comment thread src/hyperlight_host/src/hypervisor/virtual_machine/hvf/mod.rs
jsturtevant
jsturtevant reviewed Jul 31, 2026
View reviewed changes
Comment thread src/hyperlight_host/src/hypervisor/virtual_machine/hvf/mod.rs Outdated
jsturtevant
jsturtevant reviewed Jul 31, 2026
View reviewed changes
Comment thread src/hyperlight_host/src/hypervisor/virtual_machine/hvf/mod.rs
jsturtevant
jsturtevant reviewed Jul 31, 2026
View reviewed changes
Comment thread src/hyperlight_host/src/hypervisor/mod.rs
@syntactically
syntactically force-pushed the lm/hvf branch 3 times, most recently from 2484dc7 to 1f35d52 Compare August 1, 2026 17:31
@syntactically
syntactically changed the base branch from main to lm/enable-a64kvm-ci August 1, 2026 17:34
@syntactically
syntactically force-pushed the lm/hvf branch 2 times, most recently from d77b523 to 9f409ec Compare August 1, 2026 17:49
jsturtevant
jsturtevant reviewed Aug 11, 2026
View reviewed changes
Comment thread src/hyperlight_host/src/mem/shared_mem.rs Outdated
jsturtevant
jsturtevant reviewed Aug 11, 2026
View reviewed changes
Comment thread src/hyperlight_host/src/mem/shared_mem.rs Outdated
jsturtevant
jsturtevant reviewed Aug 11, 2026
View reviewed changes
Comment thread src/hyperlight_host/src/hypervisor/virtual_machine/hvf/mod.rs
jsturtevant
jsturtevant reviewed Aug 11, 2026
View reviewed changes
Comment thread src/hyperlight_host/src/hypervisor/virtual_machine/hvf/mod.rs
@syntactically

syntactically commented Aug 11, 2026

Copy link
Copy Markdown
Member Author

I like the typed error changes. Was there something particular that made you change it?

The VM errors are already typed errors, and now the HvfSyncError needs to be able to include an error from shared-memory. So, unless we want to break the original plan for transition and let the already-converted subsystem reference an anyhow error, I had to convert the shared memory subsystem that is now a dependency of the VM errors.

@syntactically
nix: Support developing on macOS
8514a34 
Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
@syntactically
Reduce duplication of interrupt handle state machines
1331771 
Previously, the Linux and WHP interrupt handle implementations used
exactly the same logic to update three bits of atomic state, and further
interrupt handle implementations would likely need the same. This commit
extracts the state machine logic into a single shared implementation
used by both interrupt handles. It also refactors
`WindowsInterruptHandle` to pull the logic around locking into another
separate type, which will be shared with other interrupt handle
implementations in the future

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
@syntactically
Abstract out interrupt handle retry logic
436292f 
Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
@syntactically
Be more careful about cfg(target_os = "linux") vs cfg(unix)
23ab9a9 
This commit moves a bunch of code (primarily mmap-based implementations
of things) from cfg(target_os = "linux") to cfg(unix), allowing it to
compile on MacOS (where it will be useful). It also moves the kvm and
mshv package dependencies from cfg(unix) to cfg(target_os = "linux"), so
that Cargo does not try (and fail) to build them on MacOS.

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
@syntactically
hvf: add cfg items and detection stub
cb28c4b 
Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
@syntactically
hvf: add Rust bindings
9344629 
Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
@syntactically
hvf: add interrupt handle using hv_vcpus_exit
2168568 
todo: squash: hvf interrupt handle implementation
Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
@syntactically
Disentangle host and guest page sizes
ed7b5e1 
On AArch64, there is no particular reason to expect the page size of the
host to match the page size of the guest, but the host previously used
some constants (`hyperlight_common::mem::{PAGE_SIZE, PAGE_SIZE_USIZE,
PAGE_SHIFT}`) both when computing guest memory layouts and when
constructing host->guest mappings.

This commit removes all the uses of those ambiguous constants, replacing
them with `page_size::get()` when the host page size is needed and
`hyperlight_common::vmem::PAGE_SIZE` when the guest page size is
required.  It also makes a few tweaks to remove assumptions that the
page sizes are the same, allowing (at the very least) operation on a
host with 16k pages while allowing guests to continue using 4k pages.

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
@syntactically
Change guest layout to support 16k host pages
b460242 
Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
@syntactically
Make VirtualMachine::set_{regs,sregs,fpu} take &mut
d4f30ae 
These conceptually mutate the state of the guest, so it makes sense that
they should take &mut.  This is useful for HVF, which stores a copy of
the regs when they are set, and in this manner can avoid an extra Rust
lock.

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
@syntactically
[tests] hvf: support identifiers and adjust for hvf
b6fb97c 
This adds new variants of the existing create_200_sandboxes test that
stress-test the behaviour of the hvf code in situations with many more
threads than vcpus. It also adjusts one test which relies on the
preservation of system register state whose preservation Hyperlight
does not guarantee; while that state is preserved on other
backends (up to a snapshot restore) it can be lost on hvf if the vcpu
is destroyed and recreated.

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
@syntactically
Restrict usage of MIDR_EL1 to Linux
2939956 
Other operating systems do not support its use in EL0

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
@syntactically
Initial single-address-space support for Hypervisor.framework
4ac8c03 
Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
@syntactically
cargo: Add codesigning runner script for MacOS
056ef17 
On MacOS, communicating with the hypervisor requires that a binary be
signed with an "entitlement" `com.apple.security.hypervisor`. This
commit adds a script that locally signs and then runs a binary, and
configures Cargo to use that script to run tests/exmaples/etc.

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
@syntactically
aarch64: Move machinery for decoding ESR_ELx to hyperlight-common
34b5876 
It will be used in hyperlight-host in the Hypervisor.framework
implementation.

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
@syntactically
ci: enable running on MacOS/HVF
daa4a5e 
Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
@syntactically
Convert mem::shared_mem module to use thiserror structured errors
644b66a 
It will be used from the hvf code, which needs to return a structured
error itself.

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>
@syntactically

syntactically commented Aug 14, 2026

Copy link
Copy Markdown
Member Author

@jsturtevant @ludfjig addressed your comments. I pushed granular commits at 6c6a304, but also pushed again the result of --autosquash so that we won't have to wait for CI + re-approval if automatically dismissed if this is approved with no changes.

@github-actions github-actions Bot removed the ready-for-review PR is ready for (re-)review label Aug 14, 2026
@ludfjig ludfjig mentioned this pull request Aug 14, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@jsturtevant jsturtevant jsturtevant approved these changes
@ludfjig ludfjig ludfjig approved these changes
@danbugs danbugs Awaiting requested review from danbugs danbugs is a code owner
@devigned devigned Awaiting requested review from devigned devigned is a code owner
@dblnz dblnz Awaiting requested review from dblnz dblnz is a code owner
@jprendes jprendes Awaiting requested review from jprendes jprendes is a code owner
@simongdavies simongdavies Awaiting requested review from simongdavies simongdavies is a code owner
@andreiltd andreiltd Awaiting requested review from andreiltd andreiltd is a code owner
@squillace squillace Awaiting requested review from squillace squillace is a code owner

Assignees

No one assigned

Labels

kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. regen-goldens Regenerate snapshot golden fixtures

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@syntactically @siennathesane @jsturtevant @ludfjig